In order to provide more complete, accurate, and timely query responses, a search engine will create indexes for all of the documents present. Many indexes will index every word in a document; therefore, the index is much larger than the size of the document. For all documents that are present within a search engine, an optimized index can be created offline, and then served.
A search engine receives fresh documents continuously. Therefore, updating the optimized index for each received fresh document would require taking the index offline constantly. In order to avoid this, small indexes of fresh documents can be created, which require much less processing time. However, serving a large number of small indexes causes performance degradation, high latencies, and less effective use of hardware. A complete index merge takes a large amount of time. In addition, a completely new index needs to be created for each change, even when the change is relatively minor.